Situational awareness components of an enhanced vision system

ABSTRACT

A virtual sphere provided by an enhanced vision system includes synthetic imagery filling said virtual sphere and a common view window mapped to a dedicated position within the synthetic imagery. Imagery of the line of sight of a user is displayed in the common view window. By providing the common view window, visual communication between all users may be possible. By connecting a virtual user to the enhanced vision system and by displaying the imagery for the line of sight of the virtual user in the common view window, the workload of a human operator may be reduced and the time line of actions may be shortened. The enhanced vision system of the present invention may be used, but is not limited to, in a military aircraft to enhance the situational awareness of the flight crew.

BACKGROUND OF THE INVENTION

The present invention generally relates to enhanced vision systems and,more particularly, to situational awareness components of an enhancedvision system and a method for enhancing flight crew situationalawareness.

The maintenance of situational awareness has always been and is stilltoday a pilot's uppermost concern. Loss of situational awareness is mostoften a main factor in airplane accidents, which is true for commercialaircraft as well as for military aircraft. One of the most dangerouschallenges military aviators face is poor visibility, especially duringoperations in unprepared environments. In recent years, enhanced visionsystems have been developed that improve the ability of pilots to seeairport features and surrounding terrain at night and during periods ofreduced visibility while flying close to the ground. Still, withconstantly increasing complexity of the tasks of a flight crew,especially of a crew flying a military plane involved in war actions,there is a need to further develop existing enhanced vision systems inorder to enhance flight crew situational awareness. For example, aflight crew of a military airplane carrying a weapon, such as theadvanced tactical laser (ATL) or a gunship, needs to maintain commonawareness for battle space while focused on individual tasks. A weaponoperator may be narrowly focused on a single target while cockpitcrewmembers may have broad situational awareness. If the cockpit crewbecomes aware of an important new development, the cockpit crew needs torapidly cue the weapon operator. Similarly, a battle manager may observean emerging threat to the aircraft and need to notify the cockpit crewimmediately, for example, by visually cueing them. Furthermore, it mayhappen that only one of the crewmembers becomes aware of a newlydeveloping situation. In this case, this crewmember needs to be able tocommunicate with the other crewmembers. Presently, there is no effectivesolution on how to share visual information among flight crewmembers,such as a weapon operator or a battle manager, to rapidly cue othermembers to targets or situations observed by one crewmember.

Furthermore, a weapon operator, for example, an ATL weapon operator,needs to maintain situational awareness of multiple targets, whilefocused on a single target at a time. Currently, the weapon operator isrequired to search for each target separately. While focused on onetarget, other targets may move under cover, making it harder for theweapon operator to search for them. Therefore, if more than one targetneeds to be tracked at the same time, prior art requires one operatorfor each target to be tracked.

Another task that requires situational awareness of the flightcrewmembers, for example, of a low flying helicopter, is obstacledetection, for example, of power lines, since striking a power line willbe disastrous for any aircraft. Passive power line detection systems foraircraft have been developed, for example, U.S. Patent Application No.U.S. 2002/0153485 A1 published by Nixon et. al. This obstacle detectionsystem determines the presence of small, curvilinear objects, such aspower lines. While the detected objects will be displayed for the pilotsuch that evasive maneuvers can be performed by the pilot as necessary,the pilot cannot change his line of sight and look to the right or left.The pilot's line of sight needs to be where he suspects, for example,power lines.

Prior artificial vision systems typically use a single turreted sensorsystem that is slaved to a pilot's line of sight. As the pilot turns hishead, the entire turret rotates to follow his line of sight.Consequently, all users can see imagery only in the pilot's line ofsight. Thus, if the pilot is looking to one side, an operator aide, suchas an obstacle detection system or a tracking system, can only viewimagery in that direction. Using prior art vision systems it is notpossible that the users can independently monitor views in differentdirections.

Currently a next generation of enhanced vision systems (EVS) is beingdeveloped, for example, the enhanced vision system described by Yelton,Bernier, and Sanders-Reed in Proc SPIE, 5424, April 2004, herebyincorporated by reference, combine imagery from multiple sensors,possibly running at different frame rates and pixel counts, onto adisplay. In the case of a helmet mounted display (HDM), the user line ofsight is continuously changing with the result that the sensor pixelsrendered on the display are changing in real time. In a prior artenhanced vision system, the various sensors provide overlapping fieldsof view, which requires stitching imagery together to provide a seamlessmosaic to the user. Furthermore, different modality sensors may bepresent requiring the fusion of imagery from the sensors having a commonfield of view. Still further, it is possible to combine sensor imagerywith synthetic imagery, such as 3D terrain from digital elevation maps,overhead satellite imagery, or flight path symbology. The output of anenhanced vision system may be presented on a head-down, head-up, orhelmet mounted display. All of this takes place in a dynamic flightenvironment where the aircraft (with fixed mounted sensors) is changingposition and orientation while the users are independently changingtheir lines of sight. Modern enhanced vision systems, for example, theenhanced vision system described by Yelton, Bernier, and Sanders-Reed,may provide new opportunities for visual sharing information and forusing independent operator aides and intelligent agents, not availablein systems pre-dating enhanced vision systems. However, current priorart enhanced vision systems are “dumb” systems in the sense that thesesystems provide integrated imagery to a human user who supplies all theintelligence for interpretation.

Prior art further includes, for example, an enhanced vision systemcalled “Flying Infrared for Low-level Operations” (FLILO) disclosed byGuell in IEEE AES Systems Magazine, September 2000, pp. 31-35. FLILOenhances situational awareness for safe low level/night time andmoderate weather flight operations, such as take-off, landing, taxiing,approaches, drop zone identification and short austere airfieldoperations. FLILO provides electronic/real time vision to the pilotsthrough a series of imaging sensors, an image processor, and a widefield-of-view see-through helmet mounted display integrated with a headtracker. While enhancing the situational awareness of a flight crew, theFLILO enhanced vision system does not offer visual communication betweenflight crew members or allows the user to maintain situational awarenessof multiple tasks.

As can be seen, there is a need for increasing not only the pilot'ssituational awareness but also the flight crew's situational awareness.Furthermore, there is a need to enable instant visual communicationbetween a cockpit crew, a weapon operator, a battle manager, or otherpersonnel of a military aircraft. Also, there is a need to share visualinformation among cockpit crewmembers. Moreover, a need exists to enablea weapon operator to maintain situational awareness of multiple targets.Still further, there is a need to add “intelligent” components toexisting enhanced vision systems that may provide help to human users orthat may replace human users.

There has, therefore, arisen a need to provide components that enablesharing of visual information among flight crew members and that may beadded to existing enhanced vision systems. There has further arisen aneed to provide components that reduce the workload of an operator, suchas a weapon operator, and that may be added to prior art enhanced visionsystems. There has still further arisen a need to provide “intelligent”components that may handle various flight and battle operations based onthe broad area coverage provided by an enhanced vision system.

SUMMARY OF THE INVENTION

The present invention provides situational awareness components ofenhanced vision systems and a method for enhancing flight crewsituational awareness. The present invention further provides a platformfor sharing visual information among flight crewmembers. The presentinvention still further provides components that combine tasks, such asautomatic target tracking, obstacle detection, or blind spot monitoring,with the broad area coverage provided by an enhanced vision system.These components may be used as, but are not limited to, operator aidesor intelligent agents to assist, for example, battle managers andpayload managers, of military aircraft carrying an advanced tacticallaser or gunship. The present invention still further provides a methodfor enhancing flight crew situational awareness.

In one aspect of the present invention, a virtual sphere provided by anenhanced vision system comprises synthetic imagery filling the virtualsphere and a common view window mapped to a dedicated position withinthe synthetic imagery. The imagery of the line of sight of a user isdisplayed in the common view window.

In another aspect of the present invention, a virtual sphere provided byan enhanced vision system comprises sensor synthetic imagery filling thevirtual sphere, sensor imagery overlaying the synthetic imagery, a firstcommon view window mapped to a dedicated position within the syntheticimagery, and a second common view window mapped to a dedicated positionwithin the synthetic imagery. The imagery of the line of sight of afirst user is displayed in the first common view window. The imagery ofthe line of sight of a second user is displayed in the second commonview window.

In still another aspect of the present invention, an enhanced visionsystem comprises a plurality of physical sensors providing sensorimagery; synthetic imagery supplementing the sensor imagery andincluding a common view window mapped to a dedicated position within thesynthetic imagery, a first user, and a second user. The first user is ahuman operator wearing a helmet mounted display. The common view windowis visible in the helmet mounted display. The human operator selects theposition of the common view window in the helmet mounted display. Thesecond user is a virtual user.

In a further aspect of the present invention, an enhanced vision systemcomprises a plurality of physical sensors providing sensor imagery;synthetic imagery supplementing the sensor imagery, a video distributionmodule, a first enhanced vision system processor receiving imagery fromthe video distribution module, a second enhanced vision system processorreceiving imagery from the video distribution module, a third enhancedvision system processor receiving imagery from the video distributionmodule, a first human operator connected with the first enhanced visionsystem processor, a second human operator connected with the secondenhanced vision system processor, and a virtual user connected with thethird enhanced vision system processor. The synthetic imagery includes afirst common view window mapped to a dedicated position within thesynthetic imagery and a second common view window mapped to a dedicatedposition within the synthetic imagery. The video distribution modulecombines the sensor imagery and the synthetic imagery. The first humanoperator wears a first helmet mounted display. The first common viewwindow and the second common view window are visible in the first helmetmounted display. The second human operator wears a second helmet mounteddisplay. The first common view window and the second common view windoware visible in the second helmet mounted display. In still a furtheraspect of the present invention, an enhanced vision system comprises aplurality of physical sensors fixed mounted to an aircraft providingsensor imagery, synthetic imagery supplementing the sensor imagery, avideo distribution module, a first enhanced vision system processorreceiving imagery from the video distribution module, a second enhancedvision system processor receiving imagery from the video distributionmodule, a third enhanced vision system processor receiving imagery fromthe video distribution module, a first human operator connected with thefirst enhanced vision system processor, a second human operatorconnected with the second enhanced vision system processor, and apassive obstacle detection device connected with the third enhancedvision system processor. The synthetic imagery includes a first commonview window mapped to a dedicated position within the synthetic imagery,a second common view window mapped to a dedicated position within thesynthetic imagery, and a third common view window mapped to a dedicatedposition within the synthetic imagery. The video distribution modulecombines the sensor imagery and the synthetic imagery. The first humanoperator is on board of the aircraft. The second human operator is offboard of the aircraft. The first human operator sends a line of sightpointing request to the first enhanced vision system processor. Thefirst human operator receives imagery for the requested line of sightfrom the first enhanced vision system processor. The imagery isdisplayed in the first common view window. The second human operatorsends a line of sight pointing request to the second enhanced visionsystem processor. The second human operator receives imagery for therequested line of sight from the second enhanced vision systemprocessor. The imagery is displayed in the second common view window.The passive obstacle detection device sends a line of sight pointingrequest to the third enhanced vision system processor. The passiveobstacle detection device receives imagery for the requested line ofsight from the third enhanced vision system processor. The imagery isdisplayed in the third common view window.

In still another aspect of the present invention, a method for enhancingflight crew situational awareness comprises the steps of: equipping anaircraft with an enhanced vision system including a common view window;connecting a first human operator with the enhanced vision system;producing line of sight imagery for the first human operator with theenhanced vision system; connecting a virtual user with the enhancedvision system; producing line of sight imagery for the virtual user withthe enhanced vision system; displaying the line of sight imagery of thevirtual user in the common view window; alerting the first humanoperator; and viewing line of sight of the virtual user by the firsthuman operator.

These and other features, aspects and advantages of the presentinvention will become better understood with reference to the followingdrawings, description and claims.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a schematic view of a virtual sphere provided by an enhancedvision system according to one embodiment of the present invention;

FIG. 2 is a schematic view of a virtual sphere provided by an enhancedvision system according to another embodiment of the present invention;

FIG. 3 is a block diagram of an enhanced vision system according to oneembodiment of the present invention; and

FIG. 4 is a flow chart of a method for enhancing flight crew situationalawareness according to one embodiment of the present invention.

DETAILED DESCRIPTION OF THE INVENTION

The following detailed description is of the best currently contemplatedmodes of carrying out the invention. The description is not to be takenin a limiting sense, but is made merely for the purpose of illustratingthe general principles of the invention, since the scope of theinvention is best defined by the appended claims.

Broadly, an embodiment of the present invention provides situationalawareness components of an enhanced vision system. Contrary to the knownprior art, the spherical coverage provided by an enhanced vision systemas in one embodiment of the present invention includes at least onecommon view window that displays the line of sight of a designatedflight crewmember. Furthermore, operator aides and intelligent agents asin another embodiment of the present invention can be added to existingenhanced vision systems as virtual crewmembers. Enhanced vision systemsincluding situational awareness components as in one embodiment of thepresent invention may be used, for example, in military aircraft toenhance the situational awareness of the flight crew. The situationalawareness components as in another embodiment of the present inventionmay be used as, but are not limited to, operator aides or intelligentagents to assist, for example, battle managers of a military helicoptercarrying an advanced tactical laser or gunship, or payload managers of atransport aircraft. By using an enhanced vision system as in oneembodiment of the present invention it may be possible that a humanuser, independent operator aides and intelligent agents, such as anobstacle detection system or a tracking system, can independentlymonitor views in different directions, which is not possible using priorart single turreted sensor systems. Furthermore, it may be possible, touse the enhanced vision system as in one embodiment of the presentinvention in ground vehicles, such as truck, ships, etc., both militaryand commercial.

In one embodiment, the present invention provides a common view windowincorporated within a virtual sphere provided by a prior art enhancedvision system. The imagery displayed in this common view window may bedesignated to the line of sight of a designated flight crewmember. Thisenables all other flight crew members to follow the line of sight of thedesignated crewmember and adjust their own line of sight if needed.Prior art enhanced vision systems do not provide components that allowcommunication between flight crew members as in one embodiment of thepresent invention.

An embodiment of the present invention further provides a plurality ofcommon view windows incorporated within a virtual sphere provided by aprior art enhanced vision system. Each of the common view windows couldbe assigned to the line of sight of one flight crewmember furtherenhancirig the communication between flight crewmembers. Contrary to theprior art, where each flight crewmember receives the various sensorimages independently and only sees images in his line of sight, it willbe possible, by adding the common view windows as in one embodiment ofthe present invention, that each crew member also follows the line ofsight of each of the other crew members. By providing a plurality ofcommon view windows as in one embodiment of the present invention, itmay further be possible that one flight crew member alerts the otherflight crew members of his line of sight and that each of the other crewmembers may switch their line of sight directly to the alerted line ofsight. Such communication between flight crewmembers is not possibleusing prior art enhanced vision systems.

An embodiment of the present invention further provides virtual users ofan enhanced vision system, such as operator aides and intelligentagents, that could assist or even replace human operators, for example,flight crew members of a military aircraft or a transport aircraft. Theimagery for the line of sight of operator aides as in one embodiment ofthe present invention, for example, automatic target trackers andpassive obstacle detection systems, may be included as part of thesynthetic vision overlay of the enhanced vision system to the humanoperator, such as a flight crew member. Contrary to the known prior art,operator aides, such as automatic target trackers, as in one embodimentof the present invention could enable, for example, a weapons operatorto track multiple targets instead of tracking only one target. Bycombining, for example, automatic target trackers with wide coveragesensors available through prior art enhanced vision systems, theexisting standard auto tracker technology can be extended and the flightcrew situational awareness can be enhanced. By providing intelligentagents, integrated imagery provided by an enhanced vision system can beinterpreted by an electronic processing unit instead of supplying allthe intelligence for interpretation of the imagery by a human user ascurrently done.

An embodiment of the present invention further provides a method forenhancing flight crew situational awareness. By providing a platformthat enables sharing of visual information obtained from a prior artenhanced vision system among flight crew members as in one embodiment ofthe present invention, the flight crew awareness can be improvedcompared to prior art methods. By adding virtual users, such as operatoraides and intelligent agents, to an existing enhanced vision system, thesituational awareness of each flight crewmember may be enhanced byreducing the workload of the human operator. Furthermore, the timelineof operations may be improved using the method for enhancing flight crewsituational awareness as in one embodiment of the present inventioncompared to prior art methods that only use a prior art enhanced visionsystem.

Referring now to FIG. 1, a schematic view of a virtual sphere 10provided by an enhanced vision system 30 to a user 20 is illustratedaccording to one embodiment of the present invention. The virtual sphere10 may include synthetic imagery 12 and sensor imagery 13. The syntheticimagery 12 may fill the entire virtual sphere 10. The sensor imagery 13may overlay the synthetic imagery 12 according to the position of fixedphysical sensors 31 (shown in FIG. 3), for example, on an aircraft.Images provided by the physical sensors 31 (illustrated in FIG. 3), suchas visible light cameras and infrared cameras, may be stitched togetherto a combined multi-spectral image viewable as sensor imagery 13 by theuser 20. The user 20 may be a human operator 21, as shown in FIG. 1 or avirtual user 23, as shown in FIG. 3. The sensor imagery 13 may furtherinclude images from modality sensors, such as an image 14 from a rearview mirror sensor. The sensors 31 providing the sensor imagery 13 maybe preferably fixed mounted sensors mounted to an aircraft.Consequently, the sensors 31 change position and orientation as theaircraft does. The synthetic imagery 12 may include a synthetic map 15created from a digital terrain database and a common view window 16. Thesynthetic imagery 12 may further include overhead satellite imagery andflight path symbology (not shown). The sensor imagery 13 and thesynthetic imagery 12 may be combined by the enhanced vision system 30(FIG. 3) to provide the spherical coverage 10. It may also be possibleto turn off the sensor imagery 13, for example, in conditions of zerovisibility, and to use the synthetic imagery 12 only. The output of theenhanced vision system 30 may be presented on a helmet mounted display17. The output of the enhanced vision system 30 may further be presentedon a head-down or head-up display (not shown). The human operator 21 maywear a helmet mounted display 17 and may change his line of sight byturning his head 211. More than one human operator 21 may be connectedto the enhanced vision system 30. Each human operator 21 may wear ahelmet mounted display 17 and may change his line of sight independentlyfrom other human operators 21. The common view window 16 may be visiblein the helmet mounted display 17. The human operator 21 may be a memberof a flight crew, such as a pilot, a cockpit crewmember, a weaponoperator, a battle manager, or a payload manager on board of anaircraft. The human operator 21 may further be a remote operator on theground for an unmanned aircraft, a human operator (off board) 22.

As shown in FIG. 1, the common view window 16 may be mapped to adedicated position on the virtual sphere 10 within the area of syntheticimagery 12. Alternatively, the human operator 21 may be able to selectthe position of the common view window 16 in his helmet mounted display17 as desired. Furthermore, the common view window 16 may be programmedto follow the line of sight of the human operator 21. Programming thecommon view window 16 to follow the line of sight of a human operator 21may enable the human operator 21 to drag the common view window 16around the virtual sphere 10 while he is trying to find the same imageryin his helmet mounted display 17 as displayed in the common view window16. The imagery displayed in the common view window 16 may be assignedto the line of sight of a human operator 21, such as a flightcrewmember, that takes control of the common view window 16.Consequently, all human operators 21 connected to the enhanced visionsystem 30, for example, all flight crewmembers may view the line ofsight of the human operator 21 that took control of the common viewwindow 16. If needed, a different human operator 21 may take control ofthe common view window 16 to alert the other users 20. The common viewwindow 16 may then display the line of sight of the human operator 21that took control of the common view window 16 visible to all users 20of the enhanced vision system 30. For example, the co-pilot of anaircraft may take control of the common view window 16. Wherever theco-pilot may look, his line of sight will be displayed in the commonview window 16 visible for all other flight crewmembers in their helmetmounted displays 17. If one of the other flight crewmembers sees aimagery 12 or 13 in his helmet mounted display 17 that he wants toshare, this flight crewmember takes control of the common view window 16and instantly, this information becomes visible in the common viewwindow 16 that may be visible in the helmet mounted display 17 of eachcrewmember. The user 20 that takes control of the common view window 16may further be a remote operator 22 (as shown in FIG. 3). The user 20that takes control of the common view window 16 may further be a virtualuser 23 (as shown in FIG. 3). It may further be desirable to add someinformation to the imagery displayed in the common view window 16, forexample, information identifying the human operator that has control ofthe common view window 16. Enabling the human operator 21 who hascontrol of the common view window 16 to draw overlays on the imagerydisplayed in the common view window 15 may further enhance the graphiccommunication between human operators 21.

Referring now to FIG. 2, a schematic view of a virtual sphere 10provided by an enhanced vision system 30 to a user 20 is illustratedaccording to another embodiment of the present invention. The virtualsphere 10 may include sensor imagery 13 and synthetic imagery 12. Thesensor imagery 13 may contain a combined multi-spectral image. Thesynthetic imagery 12 may include a synthetic map 15, a common viewwindow 161, a common view window 162, a common view window 163, and acommon view window 164. The common view window 161, the common viewwindow 162, the common view window 163, and the common view window 164may be visible in the helmet mounted display 17 of a first humanoperator 21 (on board), in the helmet mounted display 17 of a secondhuman operator 21 (on board), and in the helmet mounted display 17 of athird human operator 22 (off board). The common view window 161 maydisplay the imagery of the line of sight of a first human operator 21(on board). The common view window 162 may display the imagery of theline of sight of the second human operator 21 (on board). The commonview window 163 may display the imagery of the line of sight of thethird human operator 22 (off board). The common view window 164 maydisplay the imagery of the line of sight of a user 20. The first humanoperator and the second human operator may be on board, for example, ofan aircraft and, therefore, at the same location as the enhanced visionsystem 30. The third human operator 22 may be a remote operator that isoff board, for example, of an aircraft and, therefore, at a differentlocation than the enhanced vision system 30. The user 20 may be anadditional human operator 21 (on board), an additional human operator 22(off board), or a virtual user 23 (FIG. 3). The virtual user 23 may bean operator aide or an intelligent agent. Additional common view windows16 may be mapped on the virtual sphere 10 besides the common viewwindows 161, 162, 163, and 164 (shown in FIG. 2). The imagery displayedin the common view windows 161, 162, 163, and 164 may be assigned to aspecific user 20 as described above. It may further be possible that anyhuman operator 21 connected to the enhanced vision system 30 may takecontrol of the common view window 161 as needed.

Referring now to FIG. 3, a block diagram of an enhanced vision system 30is illustrated according to one embodiment of the present invention. Theenhanced vision system 30 may include a plurality of physical sensors31, synthetic imagery 12, a video distribution module 32, a plurality ofenhanced vision system processors 33, and a plurality of users 20. Thesensors 31 may provide images that may be stitched together to createthe sensor imagery 13 (FIG. 1). The synthetic imagery 12 may supplementthe sensor imagery 13. The video distribution module 32 may combine thesensor imagery 13 and the synthetic imagery 12 to form the virtualsphere 10 (FIG. 1). The video distribution module 32 may distribute thecombined imagery data to the enhanced vision system processors 33. Eachenhanced vision system processor 33 may receive a line of sight pointingrequest 35 from a user 20. Corresponding to the line of sight pointingrequest 35 from the user 20, the enhanced vision system processor 33 mayprovide imagery 36 for the requested line of sight to the user 20. Thequantity of enhanced vision system processors 33 connected with thevideo distribution module 32 may depend on the quantity of users 20 andmay further be equal to the quantity of users 20. The user 20 may be ahuman operator 21 that is on board, a human operator 22 that is offboard, or a virtual user 23. The virtual user 23, such as an operatoraide or an intelligent agent, may include a processor 34. The humanoperator may be a flight crewmember of an aircraft, such as a pilot, aco-pilot, a weapons operator, a battle manager, or a payload manager. Inthe case that the user 20 is a human operator 21 (on board) or a humanoperator 22 (off board), the human operator 21 or 22 may wear a helmetmounted display 17 and the human operator 21 or 22 may send a line ofsight pointing request 35 to the enhanced vision system processor 33 byturning their head 211 (FIG. 1). The enhanced vision processor 33 maythen send imagery 36 for the requested line of sight to the humanoperator 21 or 22. The received imagery 36 may be displayed on thehelmet mounted display 17 of the human operator 21 or 22. The humanoperator 21 or 22 may then provide the analysis, processing, andinterpretation of the received imagery 36. In the case that the user 20is a virtual user 23, the processor 34 may send a line of sight pointingrequest 35 to the enhanced vision system processor 33. The enhancedvision processor 33 may then send imagery 36 for the requested line ofsight to the processor 34 of the electronic processing unit 23. Theprocessor 34 may provide the processing of the imagery 36 received fromthe enhanced vision system processor 33. Further, the imagery 36received by the processor 34 may be displayed in the helmet mounteddisplay 17 of the human operator 21 or 22, for example, visible in thecommon view window 164 (as shown in FIG. 2).

The processor 34 may be an operator aide processor. Operator aides maybe, for example, a passive obstacle detection device. The processor 34may have a forward facing line of sight, may send a pointing request 35for this forward facing line of sight, and may always receive imagery 36provided by the forward facing sensors regardless of the line of sightof the human operator 21. The imagery 36 received by the processor 34may be displayed in the helmet mounted display 17 of the human operator21, for example, in the common view window 164 (as shown in FIG. 2). Theprocessor 34 may further sound an alarm if an obstacle is detected inthe flight path, the human operator 21 may change his line of sightimmediately to the line of sight of the processor 34 and take action asneeded. Furthermore, a transport aircraft equipped with the enhancedvision system 30 may have rear looking sensors 31 that may, for example,monitor cargo deployment. The processor 34 of the virtual user 23 mayreceive imagery 36 provided by these rear looking sensors 31. Theimagery 36 received by the processor 34 may be displayed in the helmetmounted display 17 of the human operator 21, for example, in the commonview window 164 (as shown in FIG. 2). The processor 34 may visuallyalert the human operator 21 using, for example, the common view window164. The processor 34 may further sound an alarm if a problem isdetected in the cargo area, the human operator 21 may change his line ofsight immediately to the line of sight of the processor 34 and takeaction as needed.

The processor 24, as shown in FIG. 3, may further be a processor of anintelligent agent. One user 20 of the enhanced vision system 30 may be atracking system. The tracking system may be a virtual user 23 that mayinclude a processor 34 for each tracker. The tracking system may be ableto track an object as the object moves relative to the aircraft. Theprocessor 34 of a tracker may send a pointing request 35 for its currentline of sight to a target. This will generate output imagery 36 centeredon the target, which the processor 34 of the tracker uses to update itsline of sight. As the target moves, the tracker may follow the targetwithin the sensor (31) boundaries. The imagery 36 of the line of sightreceived by the processor 34 of the tracker and, therefore, the positionof the moving target may be displayed on the helmet mounted display 17of the human operator 21, for example, in the common view window 164(FIG. 2). Such tracking system may be used, for example, by a weaponoperator. The weapons operator may also be a human operator 21 asillustrated in FIGS. 1, 2, and 3. The weapon operator 21 may, forexample, identify three targets. The weapon operator 21 may initiate anautomatic tracker on each of the three targets. While the weaponoperator 21 is dealing with one target, the trackers may keep track ofthe other two targets. When the weapon operator 21 is ready to deal withone of these targets, he may quickly reacquire the target even if thetarget is hidden from the weapon operator's 21 view. Consequently, thecurrently exiting auto tracker technology may be extended by using widecoverage sensors 31 of the enhanced vision system 30. Virtual users 23may further include adaptive cruise control, station keeping, and blindspot monitoring.

Referring now to FIG. 4, a flow chart of a method 40 for enhancingflight crew situational awareness is illustrated according to anotherembodiment of the present invention. The method 40 for enhancing flightdrew situational awareness may include the steps of: providing anaircraft equipped with an enhanced vision system 30 as illustrated inFIG. 3 (step 41), connecting a first human operator 21 with the enhancedvision system 30 by wearing a first helmet mounted display 17 (step 42)(as shown in FIG. 1), connecting a second human operator 21 with theenhanced vision system 30 by wearing a second helmet mounted display 17(step 43) (as shown in FIG. 1). The method 40 may further include thesteps of: sending a line of sight pointing request 35 from first humanoperator 21 to the enhanced vision system processor 33 by turning head211 and receiving imagery 36 displayed in the first helmet mounteddisplay 17 (step 44) (as shown in FIG. 3) and sending a line of sightpointing request 35 from second human operator 21 to the enhanced visionsystem processor 33 by turning head 211 and receiving imagery 36displayed in the second helmet mounted display 17 (step 45)(as shown inFIG. 3). The first human operator 21 may then take control of commonview window 16 illustrated in FIG. 1 alerting the second human operator21 (step 46). The imagery 36 for the line of sight of the line of sightof the first human operator 21 may be displayed in the common viewwindow 16 visible for the second human operator 21 (step 47). Seeing anew event occurring in his line of sight, the second human operator 21may take control of common view window 16 illustrated in FIG. 1 alertingthe first human operator 21 (step 48). The imagery 36 (as shown in FIG.3) for the line of sight of the second human operator 21 will now bedisplayed in the common view window 16 visible for the first humanoperator 21 (step 49).

The method 40 may further include the steps of: connecting a virtualuser 23 to the enhanced vision system 30 (step 51), sending a line ofsight pointing request 35 from processor 34 of the virtual user 23 tothe enhanced vision system processor 33, and receiving imagery 36 (step52) as illustrated in FIG. 3. The virtual user 23 may take control ofthe common view window 16 alerting the first human operator and thesecond human operator 21 (step 53). The imagery 36 for the line of sightof the line of sight of virtual user 23 may be displayed in the commonview window 16 visible for the first human operator 21 and second humanoperator 21 (step 54). By providing the common view window 16 in thesynthetic imagery 12 on the virtual sphere 10, for example, as in step41, visual communication between the first human operator 21 and thesecond human operator 21 may be possible. By connecting a virtual user23 to the enhanced vision system 30 as in step 51 and by displaying theimagery 36 for the line of sight of the virtual user 23 in the commonview window 16 (as in step 53), the workload of the human operator 21may be reduced and the time line of actions may be shortened. Byproviding more than one common view window 16 (as illustrated in FIG.2), the flight crew situational awareness may be further enhanced. Eventhough the enhanced vision system 30 as in one embodiment of the presentinvention has been described to be used by pilots of an militaryaircraft, it may be possible to use the enhanced vision system 30 incommercial aircrafts and, furthermore, in ground vehicles, such astruck, ships, etc., both military and commercial.

It should be understood, of course, that the foregoing relates toexemplary embodiments of the invention and that modifications may bemade without departing from the spirit and scope of the invention as setforth in the following claims.

1. A virtual sphere provided by an enhanced vision system, comprising:synthetic imagery filling said virtual sphere; and a common view windowmapped to a dedicated position within said synthetic imagery; whereinimagery of the line of sight of a user is displayed in said common viewwindow.
 2. The virtual sphere of claim 1, wherein said user is a firsthuman operator.
 3. The virtual sphere of claim 2, wherein imagery of theline of sight of said first human operator is displayed in said commonview window.
 4. The virtual sphere of claim 1, wherein said user is asecond human operator.
 5. The virtual sphere of claim 4, wherein saidsecond operator views said imagery of the line of sight of said firsthuman operator in said common view window.
 6. The virtual sphere ofclaim 4, wherein imagery of the line of sight of said second humanoperator is displayed in said common view window.
 7. The virtual sphereof claim 4, wherein said first operator views said imagery of the lineof sight of said second human operator in said common view window. 8.The virtual sphere of claim 1, wherein said user is a virtual user. 9.The virtual sphere of claim 8, wherein imagery of the line of sight ofsaid virtual user is displayed in said common view window.
 10. Thevirtual sphere of claim 8, wherein said first operator and said secondoperator view said imagery of the line of sight of said virtual user insaid common view window.
 11. The virtual sphere of claim 8, wherein saidvirtual user is an operator aide.
 12. The virtual sphere of claim 8,wherein said virtual user is an intelligent agent.
 13. A virtual sphereprovided by an enhanced vision system, comprising: synthetic imageryfilling said virtual sphere; sensor imagery overlaying said syntheticimagery; a first common view window mapped to a dedicated positionwithin said synthetic imagery; and a second common view window mapped toa dedicated position within said synthetic imagery; wherein imagery ofthe line of sight of a first user is displayed in said first common viewwindow; and wherein imagery of the line of sight of a second user isdisplayed in said second common view window.
 14. The virtual sphere ofclaim 13, wherein said first user views said first common view windowand said second common view window, and wherein said second user viewssaid first common view window and said second common view window. 15.The virtual sphere of claim 13, further including: one additional commonview window mapped to a dedicated position within said syntheticimagery; one additional user; and wherein imagery of the line of sightof said one additional user is displayed in said additional common viewwindow.
 16. The virtual sphere of claim 15, wherein said first userviews said first common view window, said second common view window, andsaid additional common view window, wherein said second user views saidfirst common view window, said second common view window, and saidadditional common view window, and wherein said additional user viewssaid first common view window, said second common view window, and saidadditional common view window.
 17. The virtual sphere of said claim 13,wherein said first user is a pilot of an aircraft; and wherein saidsecond user is a passive obstacle detection device.
 18. The virtualsphere of said claim 13, wherein said first user is a weapon operator;and wherein said second user is a tracking system.
 19. An enhancedvision system, comprising: a plurality of physical sensors providingsensor imagery; synthetic imagery supplementing said sensor imagery andincluding a common view window mapped to a dedicated position withinsaid synthetic imagery; a first user, wherein said first user is a humanoperator wearing a helmet mounted display, wherein said common viewwindow is visible in said helmet mounted display, and wherein said humanoperator selects the position of said common view window in said helmetmounted display; and a second user, wherein said second user is avirtual user.
 20. The enhanced vision system of claim 19, furtherincluding a video distribution module, wherein said video distributionmodule combines said sensor imagery and said synthetic imagery; a firstenhanced vision system processor receiving imagery from said videodistribution module; and a second enhanced vision system processorreceiving imagery from said video distribution module; wherein saidfirst user is connected with said first enhanced vision systemprocessor; and wherein said second user is connected with said secondenhanced vision system processor.
 21. The enhanced vision system ofclaim 20, wherein said first user sends a line of sight pointing requestto said first enhanced vision system processor, wherein said first userreceives imagery for the requested line of sight from said firstenhanced vision system processor.
 22. The enhanced vision system ofclaim 20, wherein said second user sends a line of sight pointingrequest to said second enhanced vision system processor, wherein saidsecond user receives imagery for the requested line of sight from saidsecond enhanced vision system processor, wherein said imagery isdisplayed in said common view window.
 23. The enhanced vision system ofclaim 19, further including: one additional enhanced vision systemprocessor receiving imagery from said video distribution module; and oneadditional user, wherein said additional user sends a line of sightpointing request to said additional enhanced vision system processor,wherein said additional user receives imagery for the requested line ofsight from said additional enhanced vision processor, and wherein saidimagery is displayed in said common view window.
 24. The enhanced visionsystem of claim 23, wherein said additional user is a human operator onboard.
 25. The enhanced vision system of claim 23, wherein saidadditional user is a human operator off board.
 26. The enhanced visionsystem of claim 23, wherein said additional user is a virtual user. 27.The enhanced vision system of claim 19, wherein said physical sensorsare fixed mounted sensors mounted to an aircraft.
 28. An enhanced visionsystem, comprising: a plurality of physical sensors providing sensorimagery; synthetic imagery supplementing said sensor imagery and whereinsaid synthetic imagery includes: a first common view window mapped to adedicated position within said synthetic imagery; and a second commonview window mapped to a dedicated position within said syntheticimagery; a video distribution module, wherein said video distributionmodule combines said sensor imagery and said synthetic imagery; a firstenhanced vision system processor receiving imagery from said videodistribution module; a second enhanced vision system processor receivingimagery from said video distribution module; a third enhanced visionsystem processor receiving imagery from said video distribution module;a first human operator wearing a first helmet mounted display and beingconnected with said first enhanced vision system processor, wherein saidfirst common view window and said second common view window are visiblein said first helmet mounted display; a second human operator wearing asecond helmet mounted display and being connected with said secondenhanced vision system processor, wherein said first common view windowand said second common view window are visible in said second helmetmounted display; and a virtual user connected with said third enhancedvision system processor.
 29. The enhanced vision system of claim 28,wherein said first human operator sends a line of sight pointing requestto said first enhanced vision system processor, wherein said first humanoperator receives imagery for the requested line of sight from saidfirst enhanced vision system processor, and wherein said first commonview window and said second common view window are programmed to followin said line of sight of said first human operator; and wherein saidsecond human operator sends a line of sight pointing request to saidsecond enhanced vision system processor, wherein said second humanoperator receives imagery for the requested line of sight from saidsecond enhanced vision system processor, wherein said imagery isdisplayed in said first common view window visible for said first humanoperator, and wherein said first common view window and said secondcommon view window are programmed to follow in said line of sight ofsaid second human operator.
 30. The enhanced vision system of claim 28,wherein said virtual user sends a line of sight pointing request to saidthird enhanced vision system processor, wherein said virtual userreceives imagery for the requested line of sight from said thirdenhanced vision system processor, and wherein said imagery is displayedin said second common view window visible for said first human operatorand for said second human operator.
 31. The enhanced vision system ofclaim 28, further including: one additional common view window mapped toa dedicated position within said synthetic imagery; one additionalenhanced vision system processor receiving imagery from said videodistribution module; and one additional user, wherein said additionaluser sends a line of sight pointing request to said additional enhancedvision system processor, wherein said additional user receives imageryfor the requested line of sight from said additional enhanced visionprocessor, and wherein said imagery is displayed in said additionalcommon view window.
 32. An enhanced vision system, comprising: aplurality of physical sensors fixed mounted to an aircraft providingsensor imagery; synthetic imagery supplementing said sensor imagery andwherein said synthetic imagery includes: a first common view windowmapped to a dedicated position within said synthetic imagery; a secondcommon view window mapped to a dedicated position within said syntheticimagery; and a third common view window mapped to a dedicated positionwithin said synthetic imagery; a video distribution module, wherein saidvideo distribution module combines said sensor imagery and saidsynthetic imagery; a first enhanced vision system processor receivingimagery from said video distribution module; a second enhanced visionsystem processor receiving imagery from said video distribution module;a third enhanced vision system processor receiving imagery from saidvideo distribution module; a first human operator connected with saidfirst enhanced vision system processor, wherein said first humanoperator is on board of said aircraft; a second human operator connectedwith said second enhanced vision system processor, wherein said secondhuman operator is off board of said aircraft; and a passive obstacledetection device connected with said third enhanced vision systemprocessor; wherein said first human operator sends a line of sightpointing request to said first enhanced vision system processor, whereinsaid first human operator receives imagery for the requested line ofsight from said first enhanced vision system processor, and wherein saidimagery is displayed in said first common view window; wherein saidsecond human operator sends a line of sight pointing request to saidsecond enhanced vision system processor, wherein said second humanoperator receives imagery for the requested line of sight from saidsecond enhanced vision system processor, and wherein said imagery isdisplayed in said second common view window; and wherein said passiveobstacle detection device sends a line of sight pointing request to saidthird enhanced vision system processor, wherein said passive obstacledetection device receives imagery for the requested line of sight fromsaid third enhanced vision system processor, and wherein said imagery isdisplayed in said third common view window.
 33. The enhanced visionsystem of claim 32, wherein said first human operator is the pilot ofsaid aircraft.
 34. The enhanced vision system of claim 32, wherein saidfirst human operator and said second human operator view said imagerydisplayed in said first common view window, in said second common viewwindow, and in said third common view window.
 35. The enhanced visionsystem of claim 32, wherein said passive obstacle detection devicevisually alerts said first human operator and said second humanoperator.
 36. A method for enhancing flight crew situational awareness,including the steps of: equipping an aircraft with an enhanced visionsystem including a common view window; connecting a first human operatorwith said enhanced vision system; producing line of sight imagery forsaid first human operator with said enhanced vision system; connecting avirtual user with said enhanced vision system; producing line of sightimagery for said virtual user with said enhanced vision system;displaying said line of sight imagery of said virtual user in saidcommon view window; alerting said first human operator; and viewing lineof sight of said virtual user by said first human operator.
 37. Themethod for enhancing flight crew situational awareness of claim 36,further including the steps of: connecting a second human operator withsaid enhanced vision system; producing line of sight imagery for saidsecond human operator with said enhanced vision system; taking controlof said common view window with said first human operator and alertingsaid second human operator, wherein said second operator is viewing saidcommon view window; and taking control of said common view window withsaid second human operator and alerting said first human operator,wherein said first operator is viewing said common view window.
 38. Themethod for enhancing flight crew situational awareness of claim 36,further including the steps of: connecting an additional virtual userwith said enhanced vision system; producing line of sight imagery forsaid additional virtual user with said enhanced vision system; anddisplaying said line of sight imagery of said additional virtual user insaid common view window.
 39. The method for enhancing flight crewsituational awareness of claim 38, further including the steps of:adding an additional common view window to said enhanced vision system;and assigning line of sight imagery of said additional virtual user tosaid additional common view window.
 40. The method for enhancing flightcrew situational awareness of claim 36, further including the step of:connecting an additional human operator with said enhanced visionsystem; producing line of sight imagery for said additional humanoperator with said enhanced vision system; and displaying said line ofsight imagery of said additional human operator in said common viewwindow.